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oampling of High Performance 
Computing (HPC) Hardware 


Intel Paragon 
150GF NCSA 
1024 CPUs Platinum 
1 TF (x86) 
ASCI Red 12 TF 
1TF 
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1TF 13 TF (IA64) 
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NOW 


Network of Workstations 


+ Pioneered the vision for clusters of commodity processors. 
2 David Culler (UC Berkeley) started early 90's 
> SunOS on SPARC Microprocessor 
> High Performance, Low Latency Interconnect 
* First generation of Myrinet 
* Active Messages 
2 Glunix (Global Unix) execution environment 
+ Brought key issues to the forefront of commodity-based computing 
2 Global OS 
Parallel file systems 
Fault tolerance 
High-performance messaging 
oystem Management 


o 


0.0 0 
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Beowulf 


wwwW.beowulf.org 


€ Definition 
2 Collection of commodity computers (PCs) 
2 Using a commodity network (Ethernet) 
2 Running open-source operating system (Linux) 


€ Interconnect 
2 Gigabit Ethernet (commodity) 
High Latency 
Cheap 
> Myrinet, Infiniband, ... (non-commodity) 
Low Latency 
OS-bypass 
Expensive 
2 Programming model is Message Passing 
+ NOW pioneered the vision for clusters of commodity processors. 
+ Beowulf popularized the notion and made it very affordable 


+ Come to mean any Linux cluster 
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Outcomes of NOW / Beowulf 


+ Clusters of PCs Popularized 

+ Allowed more people to work on parallel 
computing 

€ Almost all software components published as 
open-source 

€ Brought key ingredients of MPPs into the 
commodity space 
2 Message passing environments 
2 Batch processing systems 

€ Extremely hard to build and run 
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d. Performance Computing 
Cluster 


Public Ethernet 


Power Distribution 
(Net addressable units as option) 





Hegents 
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Other Clusters 


+ Highly Available (HA) 


2 Generally small, less than 
8 nodes 


> Redundant components 


> Multiple communication 
paths 


> This is not Rocks 
+ Visualization Clusters 
> Each node drives a display 
> OpenGL machines 
> This is not core Rocks 
> But, there is a Viz Roll 
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The Dark Side of Clusters 


+ Clusters are phenomenal price/performance computational engines 


2 Can be hard to manage without experience 
> High-performance I/O is still unsolved 


> Finding out where something has failed increases at least linearly as 
cluster size increases 


Not cost-effective if every cluster “burns” a person just for care and 
feeding 


+ Programming environment could be vastly improved 
Technology is changing very rapidly. Scaling up is becoming 
commonplace (128-256 nodes) 
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e$ The Top 2 Most Critical 
Problems 


€ The largest problem in clusters is software skew 


2 When software configuration on some nodes is different than on 
others 


> Small differences (minor version numbers on libraries) can 
cripple a parallel program 
€ [he second most important problem is adequate job 
control of the parallel process 
2 Signal propagation 
2 Cleanup 
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PRAGWA 


PACIFIC RIM APPLICATIONS AND GRID MIDDLEWARE ASSEMBLY 


Hocks (open source clustering distribution) 


WWW.rocksclusters.org 


+ Technology transfer of commodity clustering to application 
scientists (non-technical people) 
2 "make clusters easy" 


2 Scientists can build their own supercomputers and migrate up to 
national centers, or international grids, as needed 


2 Supports more than just MPI machines OptlPuter 
+ Rocks is a cluster on set of CDs (or a DVD) < 
2 Red Enterprise Hat Linux (open source, de facto standard, and free) 
> Clustering software (PBS, SGE, Ganglia, GT4, ...) 
> Highly programmatic software configuration management 
+ Core software technology for many UCSD projects 
> BIRN, CTBP, EOL, GEON, NBCR, OptlPuter, CAMERA, ... 
+ First Software release Nov, 2000 
> Began as an MPI cluster solution 
> Now builds grid resources 
> Moving towards virtualization (XEN) and other OSes (Solaris) 
Supports x86, Opteron/EM64T, and Itanium 











* 


Pm, 
d camera 
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Simple Deployment 


€ Install a frontend 


i) 
2. 
3. 


1 
2. 
3. 


4. 
5 


Three is a Magic Number 


Insert Rocks Base CD 

Insert Roll CDs (optional components) 
Answer 7 screens of configuration 
data 


Drink coffee/tea/beer (takes about 30 
minutes to install) 





e Install compute nodes: From Zero to Cluster in Just Three CDs 
Login to frontend 
Execute insert-ethers Optional Rolls 
Boot compute node with Rocks Base a Condor 
CD (or PXE) = Grid (based on NMI R4) 
Insert-ethers discovers nodes a Intel (compilers) 
Goto step 3 a Java 
d user accounts ه‎ SCE (developed in Thailand) 


€ Ad 


€ Start computing 


a Sun Grid Engine 

=» PBS (developed in Norway) 

m Areab1 (security monitoring tools) 
a Many Others ... 
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Philosophy 


+ Caring and feeding for a system 
Is not fun 
€ System Administrators cost more 
than clusters 
> 1 TFLOP cluster is less than 
$100,000 (us) 
> Close to actual cost of a fulltime 
administrator 
€ The system administrator is the 
weakest link in the cluster 


> Bad ones like to tinker (make 
small changes) 


> Good ones still make mistakes 
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Philosophy 


continued 


€ All nodes are 100% automatically 
configured 
2 Zero "hand" configuration 
2 This includes site-specific 
configuration 
+ Run on heterogeneous standard 
high volume components (PCs) 
2 Use components that offer the 
best price/performance 
2 Software installation and 
configuration must support 
different hardware 
2 Homogeneous clusters do not 
exist 
2 Disk imaging requires 
homogeneous cluster 
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Philosophy 


continued 


€ Optimize for installation 
2 Getthe system up quickly 
2 In a consistent state 
2 Build supercomputers in hours not 
months 
+ Manage through re-installation 
2 Canre-install 128 nodes in under 20 
minutes 
2 No support for on-the-fly system 
patching 
+ Do not spend time trying to issue 
system consistency 
> Just re-install 
> Can be batch driven 
+ Uptime in HPC is a myth 
> Supercomputing sites have monthly 
downtime 
> HPC is not HA (High Availability) 
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Rocks Basic Approach 


€ Install a frontend 
i. Insert Rocks Base CD 
2. Insert Roll CDs (optional components) 
3. Answer 7 screens of configuration 


Three is a Magic Number 





data 
4. Drink coffee (takes about 30 minutes 
to install) | 
e Install compute nodes: From Zero to Cluster in Just Three CDs 
1. Login to frontend 
2. Execute insert-ethers Optional Rolls 
3. Boot compute node with Rocks Base a Condor 
CD (or PXE) = Grid (based on NMI R4) 
4. Insert-ethers discovers nodes m Intel (compilers) 
5. Goto step 3 a Java 
@ Add user accounts = SCE (developed in Thailand) 


a Sun Grid Engine 

=» PBS (developed in Norway) 

m Areab1 (security monitoring tools) 
a Many Others ... 
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Minimum Requirements 


+ Frontend + Compute Nodes 
> 2 Ethernet Ports > 1 Ethernet Port 
> CDROM > 18 GB Disk Drive 
> 18 GB Disk Drive > 512 MB HAM 
> 512 MB HAM 


= Complete OS Installation on all Nodes 

= No support for Diskless (yet) 

m Not a Single System Image 

m All Hardware must be supported by RHEL 
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Rocks Users 


+ HPC wire awards 
2 2004, and 2005 
2 Competition was 
commercial 
+ User base is international 
> Still U.S. heavy 
> Europe and Asia well 
represented 
+ High Performance 
Computing community is 
eager to adopt open- 
source clustering 
solutions 
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Five Years, 
Five Awards 


In its fifth year of 


se development, Rocks 


has won its fifth 
HPCwire Award. 
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Making Clusters Easy 


ROCKS 
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Cluster Software Stack 
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Middleware 
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Common to Any Cluster 


| 


Applications 


Middleware 


Kernel 
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CN redhat 


Red Hat 


+ Enterprise Linux 4.0 
> Recompiled from public SRPMS, including errata updates (source code) 
2 No license fee required, redistribution is also fine 
> Recompiled for all CPU types (x86, Opteron, Itanium) 
> Rocks 5.0 will be based on RHEL 5.0 (Centos, or RHEL) 


+ Standard Red Hat Linux kernel 
2 No Rocks added kernel patches 


+ No support for other distributions 


2 Hed Hatis the market leader for Linux 
In the US 
* And becoming so in Europe 


2 Trivial to support any Anaconda-based system 

2 Others would be harder, and require vendor support (SuSe - 12 months work) 
€ Excellent support for automated installation 

2 Scriptable installation (Kickstart) 


2 Very good hardware detection 
5/15/08 © 2008 UC Regents 24 








Dell Invests in Red Hat 


Michael Dell puts $99.5M in Red Hat 


Billionaire chairman of No. 1 PC maker places big bet on Microsoft 


5/15/08 


competitor. 
May 10, 2005: 1:41 PM EDT 


NEW YORK (CNN/Money) - Red Hat is getting a 
$99.5 million boost from Michael S. Dell, 
billionaire founder and chairman of Dell Inc., 
according a regulatory filing. 


Through his private investment firm, MSD, Dell 
bought the largest share of $600 million in 
debentures offered by the software developer in 
January 2004, a Securities Exchange Commission 
filing showed. 


Red Hat's main product, the Linux operating system 
for PCs, is a direct competitor to Microsoft's 
Windows. The Raleigh, N.C.-based company also 
provides support services for "open source" 
technology, which is software developed by 
communities of programmers for free use. 


Dell (Research) is the nation's largest PC maker. 


Debentures are similar to bonds in that the issuer 
promises a fixed return for a stated period of time 
on the investment. 


In the case of a public company, a debenture can 
also be converted into shares or equity. m 


ERR pie 


| 120 


GEHT Lee 





Michael Dell, billionaire chairman of 
Dell Inc., has given Red Hat a $99.5M 
injection. 





Tech‏ ل 
al‏ 


D SEE MORE STORIES 


advertiser links what's this? 


Accounting Research Manager 

Find insightful interpretations on 
GAAP and Securities and 

Exchange Commission... 
www.accountingresearchmanager.ct! 





Securities Exchange Commission 
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Batch Systems 


€ Portable Batch System and Maui 7 


— 
— 


H 


© 
> 


2 


> Long time standard for HPC queuing VO 4 


systems -A 
2 Maui provides backfilling for high 


throughput P 
2 PBS/Maui system can be fragile and s 
unstable "WM TM 


" € 
n 
\ 


2 Multiple code bases: 
PBS 
* OpenPBS 
PBSPro 


* Scalable PBS sj) 
+ Sun Grid Engine 
2 Rapidly becoming the new standard 
2 Integrated into Rocks by Scalable Systems 
2 Now the default scheduler for Rocks 


2 Robust and dynamic 
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Communication Layer 


+ None 
2 "Embarrassingly Parallel” 

+ Sockets 
2 Client-Server model 
2 Point-to-point communication 

+ MPI - Message Passing Interface 
> Message Passing 
2 Static model of participants 

+ PVM - Parallel Virtual Machines 
> Message Passing 
2 For Heterogeneous architectures 
2 Resource Control and Fault Tolerance 
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Sockets are low level 


€ Sockets 
> Point-to-Point 
> N machines = (n^2 - 2 
connections 
3 KIN 8,16 10,0 5.0. A 


+ MPI/PVM 
> Shared virtual channel 
> Implementation could be sockets 
> Easier to program 
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Sockets 


+ Open an endpoint 
+ Specify IP address and port 


+ Send / receive messages 


2 If TCP, only point-to-point 
messages 


> If UDP, option of point-to-point 
or multicast (broadcast) 


@ Shutdown connection 
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High-level TCP Example 


/* 
* SERVER CODE 
*/ 


fd = socket(); 


saddr.s_addr 


= INADDR_ANY; 
saddr .port = 1234; 
bind(fd, &saddr); 
listen(fd); 


accept(fd); 
read(fd, buffer, size); 


close(fd); 





/* 
* CLIENT CODE 
wi 


fd = socket(); 


saddr.s_addr 
saddr .port 


gethostbyname(*"c0-0"); 
1234; 


write(fd, buffer, size); 


close(fd); 





© 


Challenges with Sockets 


+ TCP 
> Reliable, but byte oriented 


> Need to write code to send and receive 
packets (at the application level) 


+ UDP 
> Unreliable 
> Need to write code to reliably send packets 
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MPI 


+ Message Passing Interface 
+ De facto standard for message passing 


> Runs over many CPU architectures and many communication 
substrates 


+ There are (and were) lots of good messaging libraries 
> But, MPI is the most pervasive 


> Developed a practical, portable, efficient and flexible standard 
> In development since 1992 
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MPI 


€ Explicitly move data like sockets, but virtualizes 
the endpoints 
> Remote endpoints addressed by integer O, 1, ..., n 


+ Primitives to support point-to-point and 
broadcast 


Process 0 Process 1 Process 2 Process 3 


5/15/08 © 2008 UC Regents 33 





© 


High-level MPI Example 





MPI_Init(); 
MPI Comm rank(&my mpi id); 


Remote mpi id = 1 
MPI Send(send buffer, buf size, remote mpi id) 


MPI Recv(recv buffer, buf size, remote mpi id) 


MPI_Finalize() 





e 


Challenges with MPI 


€ If a node fails, no easy way to reconfigure 
and route around the problem 
2 Basically, your program stops 


+ Hard to manage deployment 


>network X compiler = mpi binaries 
2 Result is several versions of MPI / cluster 
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Compile 


+ MPICH with GNU Compilers and Ethernet 


Compiler 


Path 
/opt/mpich/ethernet/gcc/bin/mpicc 
/opt/mpich/ethernet/gcc/bin/mpiCC 
/opt/mpich/ethernet/gcc/bin/mpif77 


+ MPICH with GNU Compilers and Myrinet 


Compiler 
CE 
C++ 
F77: 
5/15/08 


Path 
/opt/mpich/myrinet/gcc/bin/mpicc 
/opt/mpich/myrinet/gcc/bin/mpiCC 
/opt/mpich/myrinet/g77/bin/mpif77 


© 2008 UC Regents 36 





Compile 


+ MPICH with Intel Compilers and Ethernet 


Compiler 


F90: 


Path 
/opt/mpich/ethernet/ecc/mpicc 
/opt/mpich/ethernet/ecc/mpiCC 
/opt/mpich/ethernet/ecc/mpif77 
/opt/mpich/ethernet/ecc/mpif90 


+ MPICH with Intel Compilers and Myrinet 


Compiler 
CE 
C++ 
F77 
F90 
5/15/08 


Path 
/opt/mpich/myrinet/ecc/mpicc 
/opt/mpich/myrinet/ecc/mpiCC 
/opt/mpich/myrinet/efc/mpif77 
/opt/mpich/myrinet/efc/mpif90 
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Network File System 


+ User account are served over NFS 
2 Works for small clusters (<= 128 nodes) 
> Will not work for large clusters (21024 nodes) 


2 Network Attached Storage (NAS) is better than Linux 
* Rocks uses the Frontend machine to server NFS 
٠ We have deployed NAS on several clusters 


+ Applications are not served over NFS 


2 /usr/local/ does not exist 
> All software is installed locally from RPM 
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Hocks Cluster Software 
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Optional Drivers 


+ PVFS 
2 Parallel Virtual File System 
2 Kernel module built for all nodes 
2 User must decide to enable 
+ Myrinet 
2 High Speed and Low Latency Interconnect 
> GM/MPI for user Applications 
2 Kernel module built for all nodes with Myrinet cards 
€ Video 
2 nVidia (from Viz Roll) 
€ Add your own 
2 Cluster Gigabit Ethernet driver 
> Infiniband driver 
€ Kernel Modules are dynamically built 
+ No need to manage binary Kernel Modules 
€ Burn CPU time, not human time 
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SNMP 


Enabled on all compute nodes 
+ Great for point-to-point use 
2 Good for high detail on a single end-point 
2 Does not scale to full cluster wide use 
+ Supports Linux MIB 
> Uptime, Load, Network statistics 
© Install Software 
> Running Processes 
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oyslog 


+ Native UNIX system event logger 
2 Logs events to local dist 
e /var/log/message 
* Rotates logs daily, eventually historic data is lost 
2 Forwards all message to the frontend 
+ Scalable 
2 Can add additional loghosts 
2 Can throttle verbosity of loggers 
€ Uses 
2 Predicting hardware and software failures 
2 Post Mortem on crashed nodes 
2 Debugging System startup 
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eKV 


Hemotely Interact with Installation 
> Initial kickstart 
> He-Installation 
+ Shoot-node 
> Reinstall OS and brings up eKV 
+ eKV 
> Ssh to node while it is installing 


> See the console output over 
Ethernet 


Newer versions of Rocks (4.0+) 
use VNC 
> Graphical 
> Works on headless machines 


+ 
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© Ganglia (or SCMSWeb / SCE 


Roll) 


+ Scalable cluster monitoring system 
> Based on IP multi-cast 
> Matt Massie, et al from UCB 
=  http://ganglia.sourceforge.net 
+ Gmon daemon on every node 
> Multicasts system state 
> Listens to other daemons 
> All data is represented in XML 
+ Ganglia command line 
> Python code to parse XML to English 
+ Gmetric 
2 Extends Ganglia 








juano er[auea 


ues 


111315 GI 


> Command line to multicast single metrics 
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fast in-memory 
cluster hash image 








data from other 
gmond daemons 


d 


We $ 


Cluster Multicast 
Channel 


j d 
Mn , 1 كا‎ p 
j ^ 


data from gmetric 
command line tool 
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Ganglia Screenshot 


Host Report for Tue, 18 Mar 2003 01:28:58 +0000 ( Get Fresh Data ) 





Last {hour 4) Node View 


A ; Y 
http//gangliasoureeforgemet 





Our Cluster » britannic 


britannic Overview 


britannic LOAD last hour : 








This node is up and running ا‎ 
Time and String Metrics a i i i 
Name Value O 00:40 i EE 01:20 
boottime Tue, 18 Mar 2003 00:23:20 +0000 O 1-Minute Load [B Total CPUs [i Running Processes 
gexec OFF britannic CPU last hour 
machine type ia64 
os name Linux = 
os_release 2.4.18-e.12smp : 
sys. clock Tue, 18 Mar 2003 00:25:34 +0000 R 
uptime 0 day, 1:5 
Constant Metrics 

Name Value 
cpu. aidle 97.196 a 
cpu_num 2 z 
cpu_speed 900 MHz 00: 40 01:00 01:20 
mem_total 1011568 KB B] Memory Used Bl Memory Shared E] Memory Cached 
mtu 1500 B D Memory Buffered [E Memory Swapped [Bg Total In-Core Memory 
swap. total 1048544 KB 
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SCMSWeb Screenshot 


© © © ::: SCMSWeb Monitoring Tool ::: ] 


ERS ol | A A | © http://pragma-goc.rocksclusters.org/scmsweb/ = Qy pragma goc © 


SLUSHCL 


Knowina how vour Grid works 








General Monitoring Report Document 








Tue, May 17 2005 11:45:00 -0800 
Level: PRAGMA)! --Select Unit-- 34 








Gsic-presto Pragma001 Rocks-47 











Amber! 




















Rocks56 
p 
Name / Info 1-Min Load Avg 5-Min Load Avg 15-Min Load Avg CPU System Users Nice Idle 
rn NED 0.001 0.001 0.831 0.224 0.000 8.027 
Cluster: 11 5 z S z 
Host: Pragma Grid Load Average Last Day 5 Pragma Grid Memory Last Day E 
total: 128 ES $ Z um كو رف ال‎ P. ors ean oN E: 
running: 126 | i ; 
down: 2 z z 
CPUs: 256 2 LEE e n e > 
S ES S 
i 5 Eo S 
18: 00 00: 00 06: 00 18: 00 O0: 00 06: 00 
L]1-Min Load B Total CPUs E] Nodes Bl Application Used W Cached E] Buffered [M Free 
AGE Cluster 3.060 3.060 3.040 0.239 0.194 165.667 82.889 
Ase Cluster Load Average Last Day : Ase Cluster Memory Last Day 5 
Host: 3 
total: 9 E 
running: 9 3 u - 
down: 0 E 2 5 G + 
CPUs: 18 z E 
o EE Se E ees d 04 
18:00 00: 00 06: 00 18:00 00: 00 06: 00 - 
A 
لآ‎ 1-Min Load [B Total CPUs D Nodes Bl Application Used W Cached M Buffered [M Free i 
KKK EKE IKKE EEEN CK EE 





E 
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Cluster State Management 


+ Static Information 
2 Node addresses 
2 Node types 
2 Site-specific Muse. 
configuration 
+ Dynamic Information 
د‎ CPU utilization d 
(li ganglia 
> Which nodes are 
online 
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Cluster Database 


À 
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e$ Node Info Stored In A MySQL 
Database 





ID Primary Key 

€ lf you know SQL, you can execute powerful Name Node name 
Membership Link to Mem Table 

commands CPUs Processors 
: i lu | Rack Physical Location X 
2 Hocks-supplied command line utilities are tied m Physical Location Y 


into the database 
















| ID Primary Key 
Primary Key Name Membership name 
Appliance name Appliance Link to App Table 





Graph Dir 
Graph Node 


Distribution Link to Dist Table 





> E.g., get the hostname for the bottom 8 nodes of 
each cabinet: 


Primary Key 
Distribution name 
Release Path 
Release Language 


# cluster-fork --query="Select name from nodes where rank«8" hostname 
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Software Installation 






Descriptive information to 
configure a node 





Collection of all possible 


software packages 
AKA Distribution 


Kickstart 
file 


Compute Node! | IO Server | ه‎ | 


RPMs 
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O1 
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Software Repository 





© 


Installation Instructions 











Cluster Software Management 


Software Packages Software Configuration 
+ RPMs + Tuning RPMS 
> Standard Red Hat 2 For clusters 
(desktop) packaged د‎ For your site 
HINT HU > Other customization 
> Or your own addons 
i | + XML Kickstart 
+ Rocks-dist | 
2 Programmatic System 
2 Manages the RPM Building 
TM > Scalable 


> This is the distribution 
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Building a Rocks Distribution 








Le "DNE ed 








+ Start with Red Hat 

€ Add updates, Rocks (and optional other) software 

€ Add Kickstart profiles 

+ Modify Red Hat installation boot image 

+ Resulting in a Red Hat compatible Rocks distribution 
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Kickstart 


+ Red Hat's Kickstart 
2 Monolithic flat ASCII file 
2 No macro language 


> Requires forking based on site 
information and node type. 


@ Rocks XML Kickstart 


> Decompose a kickstart file into nodes 





and a graph 
Graph specifies OO framework -— — 
Each node specifies a service and its Node Specific 
configuration Kickstart File 


2 Macros and SQL for site configuration 
2 Driven from web cgi script 
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Kickstart File Sections 


+ Main 

2 Disk partitioning 

2 Root password 

2 RPM repository URL 

3 
+ Packages 

2 List of RPMs (w/o version numbers) 

2 The repository determines the RPM versions 

2 The kickstart file determines the set of RPMs 
+ Pre 

> Shell scripts run before RPMs are installed 

2 Rarely used (Rocks uses it to enhance kickstart) 
€ Post 

2 Shell scripts to cleanup RPM installation 

2 Fixes bugs in packages 

> Adds local information 
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What is a Kickstart File? 


+ Setup & Packages (20%) 


Gal kom 

zerombr yes 

boot Made = Boca len nii S-uselm Eo 
Skipx 

auth --useshadow --enablemdb5 
al earpart Era NL 

part ivosUuM-x si zemlZs 

mat Bob Fr re ize 

part / --size 4096 

pes p opora ل‎ zc IN E dru 
Jander EN m 


Tene Sue ام‎ CN eege te cn RUS 
keyboard us 

mouse genericps/2 

timezone --utc GMT 


RECH E EE UE dg wpies Hard wod42g 0. 


Tee E 
15m Sac al Hi 
reboot 


packages 
@Base 
@Emacs 
0 GNOME 


5/15/08 


"p Ost. 


date A sE nl 
passwd: 
shadow: 

gei ODE 
Hess 
bootparams: 
ethers: 

EOF 


+ Post Configuration (80%) 


S SWI uchi Gonm Sec B BOER 
files 

files 

files 

files dns 

files 

files 


cate M/S neocon is << MES! 


server ntp.u 
server 

fudge 
authenticate 
driftfile /e 
EOF 


csd.edu 

I TELE ET 

JS 79 1 247 MEI M Erasme 10 
no 
p cran Bon fie 


/bin/mkdir -p /etc/ntp 


cat » /etc/ntp/step-tickers «« 'EOF' 


ntp.ucsd.edu 
EOF 


/usr/sbin/ntpdate ntp.ucsd.edu 


/sbin/hwcloc 
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issues 


+ High level description of software installation 
> List of packages (RPMs) 
> System configuration (network, disk, accounts, ...) 
> Post installation scripts 
De facto standard for Linux 
Single ASCII file 
> Simple, clean, and portable 
> Installer can handle simple hardware differences 
+ Monolithic 
> No macro language 
> Differences require forking (and code replication) 
> Cut-and-Paste is not a code re-use model 
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XML KICKSTART 
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It looks something like this 
oo À ° 
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Implementation 


+ Nodes 
2 Single purpose modules 
> Kickstart file snippets (XML tags map to kickstart commands) 
> Approximately 200 node files in Rocks 


Graph 
> Defines interconnections for nodes 
> Think OOP or dependencies (class, #include) 
> Asingle default graph file in Rocks 

+ Macros 


> SQL Database holds site and node specific state 
> Node files may contain «var name="state”/> tags 
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Composition 


+ Aggregate 
Functionality 

€ Scripting 
2 ISA perl-development 


2 ISA python- 
development 


> ISA tcl-development 








python- 
development 
perl- tcl- 
development development 
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Appliances 


+ Laptop / Desktop 
2 Appliances 
2 Final classes 
2 Node types 
+ Desktop ISA 
2 standalone 
+ Laptop ISA 
2 standalone 
2 Pcmcia 
+ Specify only the 
differences 
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Architecture Differences 


+ Conditional inheritance 
+ Annotate edges with target 


base 
architectures 
€ if 1386 
> Base ISA grub 
+ if 4 386, X86. 64 


1264 
> Base IsA elilo 


+ One Graph, Many CPU 
Architectures 
2 Heterogeneity becomes easy 
> Not true for SSI or Imaging 
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Putting in all together 
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Sample Node File 


<?xml verston="1.0" standalone-"no"?» 
<!DOCTYPE kickstart SYSTEM "@KICKSTART_DTD@" [<!ENTITY ssh "openssh"»]» 
<kickstart> 

<description> 

Enable SSH 

</description> 


<package>&ssh; </package> 

<package>&ssh;-clients</package> 

«package»&ssh; -server</package> 

<package>&ssh; -askpass«/package» 
«post» 


cat &gt; /etc/ssh/ssh config &lt;&lt; 'EOF' «!-- default client setup --> 
Host * 

ForwardX11 yes 

ForwardAgent yes 
EOF 


chmod o+rx /root 
mkdir /root/.ssh 
chmod o+rx /root/.ssh 


</post> 
</kickstart>> 
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Sample Graph File 


<?xml version="1.0" standalone="no"?> 


<graph> 


</graph> 


NPACI Rocks. 


"Scripting"/» 

"Ssh"/» 

"SSL"/> 

grub"  arch="1386"/> 
"elilo" arch="1a64"/> 


"base"/» 
"accounting" /» 


"Slave-node" to="node"/> 
"Slave-node" to="nis-client"/> 
"Slave-node" to="autofs-client"/> 
"Slave-node" to="dhcp-client"/> 
"Slave-node" to-"snmp-server"/» 
"Slave-node" to="node-certs"/> 
"compute" to-"slave-node"/» 
"compute" to-"usher-server"/» 
"master-node" to="node"/> 
"master-node" to="x11"/> 
"master-node" to="usher-client"/> 


«description» 
Default Graph for 
</description> 
«edge from="base" to- 
«edge from="base" to- 
«edge from="base" to- 
«edge from="base" to- 
«edge from="base" to- 
«edge from="node" to= 
<edge from="node" to= 
<edge from= 
<edge from= 
<edge from= 
<edge from= 
<edge from= 
<edge from= 
<edge from= 
<edge from= 
<edge from= 
<edge from= 
<edge from= 
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CLUSTER SQL DATABASE 
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Cluster State Management 


+ Static Information 
2 Node addresses 
2 Node types 
2 Site-specific Muse. 
configuration 
+ Dynamic Information 
د‎ CPU utilization d 
(li ganglia 
> Which nodes are 
online 
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Nodes and Groups 


13 QO0:50:Sb:eQ:42:df compute-0-4 Remote Management 
DTF Compute 


Nodes Table e hi web Portal 
d NFS Server 


Memberships Table 
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ID MAC Name Membership Hardware Rack Rank IP Le 
1 frontend-0 1 ü ü B Le betel 
2 O0:30:¢1:d8:59:00 network 1-0 6 ü 1 لا‎ 10.255.255.254 
3 00:01:e7:1a:be:00  networco-0 6 ü ü U 10.795.205.203 
4 üO0:30:c1:d89:ac: S0  networnk-3-0 
5 O00-50:8b:a5:dd-b1 nfs0-0 ID Name Appliance Distribution 
6 00:50:8b:65:63:72 nfs0-1 iu aus 
2 dompute 2 

7 00:50:$b:35:57:f nfs 1-0 "24 OES VO Node 3 
Ss O00:50:8b:a5:40:14 —nfs-1-1 Compute with PVFS 4 
H 00:50:5b:e0:3a:a? compute-0-0 Laptop 5 
10 00:50:3b:e0:44:5e | compute-O- 1 Ethernet Switches Fi 
11 O0:50:Sb:e0:40:05 compute-0-2 Myrinet Switches 6 
12 00:50:8b:e0:40:93 compute-0-3 Power Units 7 

8 

J 


-— À 
- C 








Groups and Appliances 















mana q er 
Memberships Table uH 
portal 
nfs 


default 
default 
default 


portal 
nfs 


Appliances Table 
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ID Name Appliance Distribution Compute 
1 Frontend m minm 1 no 
2 Compute Ee $ 
3 PVFS VO Node ege EX Em BERE E = 
4 Compute with PVFS 4 == نطب لايك‎ = 
5 Laptop 5 m ala Uo ntend f default frontend 
g EE 

6 Ethernet Switches 6 2 „compute detault compute 
7 Myrinet Switches = -4- t =" puts default pufsio 

` D 
S Power Units SS 4 comp-pvfs default compute-pvfs 

للا 
Remote Management er E 5 laptop default laptop‏ 9 
DTF Compute E | & _ network‏ 10 
Web Portal 10 d 1‏ 11 
„power‏ 7 

12 NFS Server 11 


dtf- compute 
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Simple key - value pairs 


ID Membership 


= © تن تن‎ A بن نت‎ B0 مح زم‎ 
oO OO OO OO OO OC 


service 
Kickstart 
Kickstart 
Kickstart 
Kickstart 
Kickstart 
Kickstart 
Kickstart 
Kickstart 
Kickstart 
Kickstart 
Kickstart 


Component 
PublicNTPHost 
ZeroMBR 
PrivatekickstartC Gl 
PublicNetm ask 
PublicNetwo tk 
PrivateNIS Master 
PrivateHostname 
PrivatelPF onwarding 
Private Gateway 
PublickickstartB ase dir 
Lang 


Value 
ntp.ucsd.edu 
yes 
kickstart.c gi 
255.255.255.0 
182.31.21.0 
frontend-0 
frontend-0 
ves 
10.1.1.1 
install 
en_US 


+ Used to configure DHCP and to customize appliance 


kickstart files 
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Nodes XML Tools: <var> 


+ Get Variables from Database 
> «var name-"Kickstart PrivateGateway"/» 
> «var name-"Node Hostname"/» 





10.1.1.1 
compute-0-0 








2 Can grab any value from the app globals 
database table 
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e 


Nodes XML Tools: «eval» 


+ Do processing on the frontend: 
> «eval shell-"bash"» 


€ [o insert a fortune in the kickstart file: 





ETAPE "Been through Hell? 
/usr/games/fortune | 
DOGZAA Whaddya bring back for 
me?" 
-- A. Brilliant 
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Nodes XML Tools <file> 


+ Create a file on the system: 


«file name=”"/etc/hi-mom” mode-"append"» 
How are you today? 
«/file» 


+ Used extensively throughout Rocks post sections 
> Keeps track of alterations automatically via RCS. 





«file name- /etc/hi" perms- 444^» RCS checkin commands... 
How are you today? cat > /etc/hi << ‘EOF’ 
| am fine. How are you today? 
</file> | am fine. 
EOF 





chmod 444 /etc/hi-mom 
| ...RCS cleanup commands... | 
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e 


Fancy «file»: nested tags 


«file name="/etc/hi'> 


Here is your fortune for today: 


«eval» 
date + 9od-9ob-9oY" 
echo 6699 


/usr/games/fortune 
</eval> 


</file> 


5/15/08 








...RCS checkin commands... 
cat > /etc/hi << ‘EOF’ 


Here is your fortune for today: 
13-May-2005 


"Been through Hell? Whaddya 
bring back for me?" 
-- A. Brilliant 


EOF 
...RCS cleanup commands... 
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Nodes Main 


+ Used to specify basic configuration: 
2 timezone 


2 mouse, keyboard types 
2 install language 


+ Used more rarely than other tags 
+ Rocks main tags are usually a straight translation: 





«main? B. 
timezone America/Mission Beach 


«timezone»America/Mission Beach 


</timezone> rootpw --iscrypted sndk48shdlwis 
mouse genericps/2 
</main> url --url http://10.1.1.1/install/rocks-dist/.. 
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Nodes Packages 


+ <package>java</package> 
2 Specifies an RPM package. Version is automatically determined: take 
the newest rpm on the system with the name ‘java’. 
+ «package arch-"x86 64”>java</package> 
> Only install this package on x86. 64 architectures 
+ «package arch="i386,x86 64”>java</package> 


<package>newcastle</package> packages 
<package>stone-pale</package> newcastle 

<package>guinness</package> stone-pale 
guinness 
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Nodes Packages 


+ RPM name is a basename (not fullname of 


RPM) 
2 For example, RPM name of package below is ‘kernel 


# rpm -qip /home/install/rocks-dist/lan/i386/RedHat/RPMS/kernel-2.6.9-22.EL.1686.rpm 
Name : kernel Relocations: (not relocatable) 

Version : 2.6.9 Vendor: CentOS 

Release | :22.EL Build Date: Sun 09 Oct 2005 03:01:51 AM WET 
Install Date: (not installed) Build Host: louisa.home.local 

Group : System Environment/Kernel Source RPM: kernel-2.6.9-22.EL.src.rpm 


Size : 25589794 License: GPLv2 

Signature : DSA/SHAT, Sun 09 Oct 2005 10:44:40 AM WET, Key ID a53d0bab443e1821 
Packager : Johnny Hughes <johnny@centos.org> 

oummary  :the linux kernel (the core of the linux operating system) 

Description : 

The kernel package contains the Linux kernel (vmlinuz), the core of any 

Linux operating system 
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Nodes Post 


ntp-client.xml 


<post> 


/bin/mkdir -p /etc/ntp 


/usr/sbin/ntpdate «var name="Kickstart PrivateNTPHost"/> 
/sbin/hwclock --systohc 


</post> 








“post 


/bin/mkdir -p /etc/ntp 
/usr/sbin/ntpdate 10.1.1.1 
[sbin/hwclock --systohc 
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PUTTING IT TOGETHER 
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W Space-time and HTTP 


Node Appliances : Frontends/Servers 


HTTP: 
Kickstart URL (Generator) can be anywhere 


" Package Server can be (a different) anywhere 
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E Gathering Node Files 


Frontend = — 
Root = 


+ Traverse a graph to build up a 
kickstart file 


+ Makes kickstart file building 
flexible 


€ Easy to share functionality 
between disparate node types 


Compute RW 
Root 
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Another Look at XML 


«graph» 


«edge Tromm client 


«to arch="1386,x86 64^»grub-client«/to- 

<to>autofs-client</to> 

<to>installclass-client</to> 
</edge> 


</graph> 
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Partial Ordering 


«graph» 
<order head="autofs-client” tail="client”/> 
«edge from="client’> 

«to arch="1386,x86_64”>grub-client</to> 

<to>autofs-client</to> 

<to>installclass-client</to> 
</edge> 






</graph> 
€ Forces autofs-client <post> section to run before client's <post> section 
+ In order graph traversal enforces a partial ordering 


€ Applying standard graph theory to system installation 
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Application Layer 





+ Rocks Rolls 

2 Optional component 

2 Created by SDSC 

2 Created by others yi 
+ Example 

> Bio (BLAST) 

> Chem (GAMESS) 

2 Visualization Clusters 
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Inheritance and Rolls 


BUILDING ON TOP OF ROCKS 
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How Rocks is Built 


€ Rocks-dist 
2 Merges all RPMs 
* Hed Hat 
* Hocks 


> Resolves versions 
> Creates Rocks 


+ Rocks distribution 
> Looks just like Red Hat 
> Cluster optimized Red Hat 


en RIE 
Red Hat Rocks Other 
Updates RPMS RPMS 
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dms How You Create Your Own 
HOCKS 


€ Rocks-dist 


2 Merges all RPMs 
* Hocks 
* Yours 





2 Hesolves versions 
> Creates Rocks++ 


€ Your distribution 


2 Looks just like Rocks 


2 Application optimized 
Hocks 
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Extension Through Inheritance 


+ UCSD/SDSC Rocks 
BIRN 

GAMESS Portal 
GEON 

GriPhyN 

Camera EC 
Optiputer 

+ Commercial 


2 Scalable Systems 
2 Platform Computing 
+ Can also override existing 
functionality 
2 Rocks without NFS? 
2 Rocks for the desktop? 


Flow of Software 





OO 0 0 0 0 
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Need Better Flexibility in Stack 


€ Issues 


= Static Stack PICK PACKAGES 
١ Cannot redefine — 


> COMBO #3: COLD WEATHER 


* Cannot extend 
> Monolithic Stack 
* Cannot “opt out” 
* All or nothing solution == 
* E.g. PBS not SGE es 
+ What we need 


> Dynamic Stack 


THE SPORT PACKAGE WILL ADD: 


=> Com ponent Based Stack Dynamic stability control (DSC), bonnet stripes, xenon headlamps with 


powerwashers, front fog lamps, 17-inch alloy S-lite wheels with 205/45 R17 


performance or all-season run-fiat tires. 


> User / Developer Extensible 
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Rolls Break Apart Rocks 





Rolls: Modifying a Standard System Installer to Support User-Customizable Cluster Frontend Appliances. Greg Bruno, Mason J. Katz, 
Federico D. Sacerdoti, and Phil M. Papadopoulos. /EEE International Conference on Cluster Computing, San Diego, California, Sep. 2004. 
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Rocks is What You Make it 


+ Motivation 
> “Fm concerned Rocks is becoming everything for everyone" - rocks mailing list 
> “Building a cluster should be like ordering a car. | want the sports package, but not the leather seats, ...” - 
24 owning rocks developer 
> We need to let go of Rocks but hold onto the core 
Hecruit more external open-source developers 
Only trust ourselves with fundamental architecture and implementation 
> We wanted to move the SGE but need to still support PBS 


+ Rolls 
> Optional configuration and software 


> Just another CD for installed (think application pack) 
>  SGE and PBS are different Rolls 
User chooses scheduler 
PBS Roll supported by Norway 
SGE Roll supported by Singapore (and us) 
> Rolls give us more flexibility and less work 
€ Rocks is done 
> The core is basically stable and needs continued support 
> Rolls allow us to develop new ideas 
> Application Domain specific 


+ IEEE Cluster 2004 - “Rolls: Modifying a Standard System Installer to Support User-Customizable Cluster Frontend Appliances” 
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Holls are sub-graphs 


€ À graph makes it easy to 'splice' in new nodes 


+ Each Roll contains its own nodes and splices 
them into the system graph file 


graphs nodes Makefile src 


default sge-client.xml ganglia globus sge use 
sge-globus.xmli 
sge-server.xml 


sge.xml src code src code src code 


sge.xml 
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STARTING FROM THE EMPTY 
SET 
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Simplified Example 


(base, hpc, sge, bio} 
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Two different Clusters 
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Where are the Scaling Limits? 


€ Time for Kickstart Generation 
> 3-4s/host 
> O(n) 


+ Time to Download Packages 
+ Rocks uses HTTP to transport Packages 
+ Linux easily serves HTTP files at 


>  100MB/sec o 1Gbit 
2 12 MB/Sec@100Mbit 


+ Time = <#nodes> * «total MB packages» / HTTP Speed 
> Total Packages ~ 350MB 


TTT 128 Nodes 1024 Nodes 





100 Mbit 3700s (1hr) 
460s (8 min) 
© 2008 UC H 
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Avalanche installer 


+ Install nodes from a Rocks Frontend 
peer-to-peer package DB Moser 
Cac h e Repository 

+ Takes advantage of Configuration 


switched networks to 
unload the frontend 
€ Kickstart generation 

IS split between 

frontend and nodes 

€ Backoff mechanisms | 

keep the frontend Zeg —Ó ms 
Package Package Package 

load under control 

Mini-HTTPD Mini-HTTPD Mini-HTTPD 


Peer-to-Peer Package Cache 


2nd Pass 2nd Pass 
Configuration Configuration 


BitTorrent-Aware BitTorrent-Aware BitTorrent-Aware 
Package Downloader Package Downloader Package Downloader 





+ Zero administration 


Appliance Appliance Appliance 
while in install mode while in install mode while in install mode 
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€» Pre-Avalanche 


-Space-Time and HTTP 


Node Appliances : Frontends/Servers 


Kickstart URL (Generator) can be anywhere 


Package Server can be (a different) anywhere 
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€» Avalanche 


-= Space-Time and HTTP 


Node Appliances : Frontends/Servers 


Installing node receives XML of graph traversal 
Installing node converts XML into a kickstart file 


Any node can server packages over BT 
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À Glimpse at Performance 


Overview of Meteor 





+ 45 Nodes - 100 Mbit 





Meteor Load last hour > Old and Slow! 
i ` > 350MB (Slim Compute Node) 
S PP TT J i D Pre-avalanche: 
3 num ui. 1 2 Estimate: 1600s 
a+ RER CT mS ER Ue ENN 
O 1-min Load [E] Modes Bg CPUs [B Running Processes C 
+ Avalanche: 
Meteor Netvork last hour > Estimate: 900s 
> Actual: 1000s 
A € Avalanche is significantly 
quicker — and reduces load on 
the frontend 
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Avalanche 
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OptlPuter 
CallT/2 
EVL / UIC 


OPTIPORTAL 


VIZ ROLL 
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2z œ 
Sem 
= = = e 
- E fe 
= كه‎ dio 
le. E> e PA 
e$ -— <> , EI ES 
=> m æ 
e E = 
aa © 
— ==> 
ED تحت‎ 
EH => 
= 
= m 
aen ` > Y 
م‎ © 
—X = 
> ES 
Let © 
m 
enz? © 
> 
> 1 
> e = = 
> © 
e - > > 
fea: © o 2 > 0 => 
EI + H ( pd E 
> => e © 
æ P "e o 
— ba 
E? oe 
E = 
ep, = 
EH 
E 
= 
EH 
| om 
w m 
LE 
© 


5/15/08 © 2008 UC Regents 


108 





Early Work: NCSA 


+ LCD Cluster 
2 Custom framing 
5 One PC /tile 
2 Portable (luggable) 
> SC 2001 Demo 


+ NCSA Software 


> Pixel Blaster 

> Display Wall In-A-Box 
٠ OSCAR based 
* Never fully released 
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NCMIR 


+ Using Rocks 
Hand configured a 
visualization cluster 


+ “Administered the 
machine to the point 
of instability” 

- David Lee 


€ Automation is needed 
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COTS Vis: GeoWall 


€ LCD Clusters 

One PC / tile 

Gigabit Ethernet 
Optional Stereo Glasses 
Portable 


OO 0 0 0 


€ Applications 
2 Large remote sensing 
> Volume Rendering 
> Seismic Interpretation 
Brain mapping (NCMIR) 


€ Electronic Visualization 
2 Jason Leigh (UIC) 


o 


5/15/08 


Commercial Frame (Reason) 


Lab 
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er = 


OptlPortal (SAGE) 
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OptlPortal 
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Use of OptlPortal 
to Interactively View Microbial Genome (CGView) 


Acidobacteria 
Bacterium Ellin345 
(NCBI) 


Soil Bacterium 5.6 Mb 














Acis c Aci 
ا‎ 1233 Acid 





AGREE ium Ellin345 (NCBI) 
PETE Source: Raj Singh, UCSD Soil GEGE REGENTS IF 


OptlPuter 


Genomic Map (cgview) 
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A Meta-Genomic / Bio-Informatic Compute and Data Infrastructure 


CAMERA 
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CAMERA: Community Cyberinfrastructure 
for Advanced Marine Microbial 
Ecology Research and Analysis 


National LambdaRail 
Direct Connect 
Computation and Storage Complex 


PI Larry Smarr 


Joint Partnership of: 


Announced 17 Jan 2006. Public Release 13 March 2007 
$24'5We Over Seven Years © 2008 UC Regents 





J. Craig Venter MARINE 


RÉ Ee E, KE MICROBIOLOGY 
UNS "PE T WT E INITIATIVE 


Marine Genome Sequencing Project — 2 MOORE 


Measuring the Genetic Diversity of Ocean Microbes 
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DNA Basics for Non-Biologists 


Thymine 


Adenine 





" M3 y- 
GPA 
ER) Y 


backbone  - X Od 
PA 


3' end Cytosine ys 


Guanine 5' end 





€ Nucleotide bases of DNA 
2 ACTG (Adenine, Cytosine, 
Guanine, Thymine) 
> A Sequence of Bases Forms One 
Side of a DNA Strand 
2 Complementary Bases form the 
other side of DNA 
* Amatches T (pair) 
* C matches G (pair) 

+ During cell replication, DNA is 
"unzipped" . The complementary 
side can then be replicated 
perfectly 


€ Human DNA is about 3 billion 
base pairs on 23 Chromosomes 
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Sequencers Generate FASTA 





>JCVI READ 299 /library idZJCVI LIB GS-00a-01-01-2P5KB-T13532 
Atemplate id-JCVI TMPL SHAAO001 /sequencing_direction=forward 

/sample idZJCVI SMPL. 1103283000001 /clr range beginz85 /clr range end-969 /full_length=969 
GCGGTTTGGAAGGAACTTCTATTCAGAAAACAAGATAATTTATTTCAATAGGCAAATATATTATCC 
AAGAGGTAAAGTCCTTTGTGGCTCTGGCTCAATAAATGCAATGGTCTATGCAAGAGGATTAGAA 
ACAGATTATGAGAATTGGGGCACCAATAAGGAATGGAGTTTTGAAAATATAAAAAAAATATACAG 
ATCTATGGAGCAACAAATAAATGATGATAAAGAATTTCTTACAAAAGAAAAGATTCCAGTAAATAA 
TGTAAGTAAGCATCATCATCCAATTTTAGAATATTTTTTTAATGCTAGTAATGAAATTOG .... 
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Bases Amino Acids 


€ Triplets of nucleotide bases are called codons and define amino 
acids. 
2 Amino acids are the basic building blocks of proteins 
2 [here are 20 amino acids, but 4^8 = 64 nucleotide combinations. 
2 Many amino acids have multiple codons 
2 Special codons (called start and stop codons) assist in DNA translation 
during cell replication. 
€ Ambiguity in codon interpretation 
2 Depends of where you start 


> For example: GGGAAACC could be: 
* GGG, AAA, CCC (Glycine, Lysine, Proline) 
* CCA, AAC (Glycine, Asparagine) 
* CAA, ACC (Glutamic Acid, Threonine) 
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e Open Reading Frame > 
Protein 


+ Open Reading Frame (ORF) 


2 portion of an organism's genome which contains a sequence of bases 
that could potentially ^ ` a protein 
٠ ATG isa DNA start Codon, 
* An ORF has no stop Codon (TAA, TAG, TGA) 





+ When processing a raw DNA read from a Gene Sequencer, you do 
not know which is the correct reading frame. 


> For double-stranded DNA — one of 6 starting positions (3 forward, 3 
reverse) 
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La sequence Analysis and 
Comparison 


+ Raw reads are possible fragments of genes 

+ Determining the proper reading frame is difficult 

+ Fragment reads are assembled into possible 
genes (or a complete Genome) 


2 Shotgun Sequencing, DNA strand is broken up 
randomly into numerous small segments 


2 Fragments are then recombined (assembled) to 
build a whole genome 
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Sequence Search and 
Alignment 


+ BLAST (Basic Local Alignment Search Tool) 
2 Finds regions of local similarity between sequences. 


2 Compares nucleotide or protein sequences to sequence databases and calculates the 
statistical significance of matches. 


2 Can be used to infer functional and evolutionary relationships between sequences 
€ How well do the following sequences align with each other? 


TTGGAGTATCTAAACATCCAATCATATTCATTAAAGTCGATTTTCCAGAACCAGAG 
GGGCCCATAATGAAATATATTCATTTTGATTTATAGTCAAATCGACTCCATCCAAA 
GCGCGAACT 


TTGGTGTATCAAGACAACCAATCATATTCATTAATGTTGATTTTCCAGAACCGGAA 
GGCCCCATAATAGAAATATATTCATTATGATTAATATTTAAATCAACTCCATCTAAAG 
CCCGAACT 


+ BLAST statistically matches a query sequence against a database of sequences 


5/15/08 © 2008 UC Regents 126 


we ESTNE 





5 J 


MARINE 


MICROBIOLOGY 
INITIATIVE 


Cordes aad Sorte 
29 MOORE 





Moore Microbial Genome Sequencing Project 
oelected Microbes Throughout the World's Oceans 
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“| North Atlantic 
| Proteobacteria: Oligobacter sp. 
| Proteobacteria: Sphingomonas sp. 
i Proteobacteria: Loktanella vestfoldensis 
اي‎ | Proteobacteria: Photobacterium sp. 
NE U.S. Coastal Waters MAL P. 
Cyanobacteria: Microcoleus chthonoplastes 9 — ef 
Cyanobacteria: Synechococcus sp. NW Atlantic 
Nitrospira: Nitrospira marina | Proteobacteria: Erythrobacter s 
7”, |Proteobacteria: Arcobacter sp. LÍ i | P. E 
Proteobacteria: Vibrio alginolytic us d 7 i. 
Proteobacteria: Vibrio splendidus 


SE U.S. Coastal Waters 1 ps apr : 

Chloroflexi: Dehalobium sp. rchea. Pyrolobus sp. i 

Proteobacteria: Roseobacter sp. EM oom à bermudensis 
Proteobacteria: Roseovarius sp. ef Cyanobacteria: Prochlorococcus sp. Proteoba سحو‎ Ze ege eet 
Proteobacteria: Sagittula stellata Cyanobacteria: Synechococcus sp. 


i f dg "e Proteobacteria: Oceanicola granulosus 
Proteobacteria: Sulfitobacter sp. | Proteobacteria: Caminibacter sp. 9 i i 
É Proteobacteria: Sulfitobacter sp. Proteobacteria: Oceanicola batsensis 


Proteobacteria: not yet validly described 
Proteobacteria: Roseovarius sp. 
Proteobacteria: Oceanicaulis aliexandrii 
Proteobacteria: Erythrobacter literalis 
Bacteroidetes: Croceibacter atlanticus 
Bacteroidetes: Robiginitalea biformata 
Actinobacteria: Janibacter sp. 


Caribbean Sea 
Proteobacteria: Roseovarius nubinhibens 
Cyanobacteria: Gloeothece sp. 


Gulf of Mexico 
Cyanobacteria: Synechococcus sp. |. A 


| Cyanobacteria: Synechococcus sp. [^77 Western Tropical Atlantic Eastern Tropical Atlantic 


Firmicutes: Bacillus sp. Proteobacteria: Nitrosococcus oceani Proteobacteria: Nitrobacter sp. | 
Firmicutes: Bacillus Sp. | Nitrospira: Nitrospina gracilis Cyanobacteria: Synechococcus sp. 
Proteobacteria: Vibrionaceae sp. Cyanobacteria: Trichodesmium thiebautii y 


8 © 2008 UC Regent | 
www.moore.org/microgenome/worldmap.asp 








New Application of Shotgun 
oequencing 


€ Shotgun sequencing is typically performed on single, known organism 
2 DNA of that organism is cloned for many samples 
> It may be difficult/impossible to isolate the DNA of a single microbe 
Venter Institute applying to whole microbial communities 
> AFiltered water sample contains many microbes. 
This community of microbes is freeze-dried, and sent to Lab 
This community is then sequenced 
> Each individual microbe genome O(2M) base pairs 
> A Read is ~1000 base pairs, it is unknown 
* Where in the genome 
Which microbe is represented 


Sequencing a Microbial community is believed to be about as complex as building the 
human Genome 


It is critical to record the environmental conditions for further comparison 
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Some CAMERA Goals 


+ Provide an infrastructure where scientists from around the world 
can perform analysis on genetic communities 
2 Global Ocean Sampling (GOS) is the initial large data set 
* ~ 8.5 Billion base pairs of raw Reads 
2 Metadata is available for samples 
* Saline, Temperature, Geographic Location, Water Depth, Time of Day ... 
* Other metadata will be correlated with samples (e.g. MODIS Satellite) 
€ Allow others to search and compare input sequences against 
CAMERA data. 


+ Overall provide a resource dedicated to metagenomics 
2 Support new datasets 
2 Support new analysis tools and web service 


5/15/08 © 2008 UC Regents 129 


J. Craig Venter 


DN X XE € CAL rb 


© Global Ocean Survey (GOS) 
Sequences are Largely Bacterial 


~5.6 Million GOS Sequences 


A B 
i Viruses 
Viruses Archae Archae 
70, D, nr 3.1% D 7%, 





Bacteria 
28% 





Eukaryota Bacteria 
% 90.8% 
~3 Million Previously Known Sequences 


PTE Source: Shibu Yooseph etal (PIGS Biology in press 2006) IT 





Meta Data 


ID Sample Loca: Country 

JCVI SITE GS Sargasso Sea, Bermuda (UK) 
JCVI SITE GS Sargasso Sea, Bermuda (UK) 
JCVI SITE GS Sargasso Sea, Bermuda (UK) 
JCVI SITE GS Sargasso Sea, Bermuda (UK) 
JCVI SITE GS Sargasso Sea, Bermuda (UK) 
JCVI SITE GS Sargasso Sea, Bermuda (UK) 
JCVI SITE GS Sargasso Sea, Bermuda (UK) 
JCVI SITE GS Sargasso Sea, Bermuda (UK) 
JCVI SITE GS Sargasso Sea, Bermuda (UK) 
JCVI SITE GSGulf of Maine USA 

JCVI SITE GS Browns Bank, Canada 

JCVI SITE GS Outside Halifa: Canada 

JCVI SITE GS Bedford Basin, Canada 

JCVI SITE GS Bay of Fundy, Canada 

JCVI SITE GS Northern Gulf Canada 

JCVI SITE GS Newport Harb USA 

JCVI SITE GS Block Island, TUSA 

JCVI SITE GS Cape May, NJ USA 

JCVI SITE GS Delaware Bay, USA 

JCVI SITE GS Chesapeake B USA 

JCVI SITE GS Off Nags Head USA 

JCVI SITE GS South of Charl USA 

JCVI SITE GS Off Key West, USA 
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Date mm/ dd Time 


2/26/03 
2/26/03 
2/26/03 
2/26/03 
2/25/03 
2/25/03 
5/15/03 
5/15/03 
5/15/03 
8/21/03 
8/21/03 
8/22/03 
8/22/03 
8/23/03 
8/25/03 
11/16/03 
11/17/03 
11/18/03 
11/18/03 
12/18/03 
12/19/03 
12/20/03 
1/8/04 
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32 
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30 


28 
12 
25 


Location 
31°32'06"n; 6 
31°32'06"n; 6 
31°10'30"n; 6 
31°10'30"n; 6 
32910'29.4"n; 
31°32'06"n; 6 
32910'00"n; € 
32910'00"n; € 
32910'00"n; € 
42930'11"n; € 
42951'10"n; 
4498'14"n; 6 
44941'25"n; € 
4596'42"n; 6 
43°37'56"n; 
41°29'9"n; 7 
41°5'28"n; 7 
38°56'24"n; 
39°25'4"n; 7! 
38956'49"n; , 
3690'14"n; 7! 
32930'25"n; , 
24929'18"n; t€ 
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J. Craig Venter 


CADETS. 


er 
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SAN DIEGO SUPERCOMPUTER CENTER 


g Calit2’s Direct Access Core Architecture Craig Venter 


GR x "FE E WS ER 


CAMERA's Metagenomics Server Complex 





Traditional 
User 


A 


Dedicated 
oargasso Sea Data 





Compute Farm | 
(1000s of CPUs) ظ‎ 





Sorcerer || Expedition 
(GOS) 





« Request Z 


JGI Community 
oequencing Project 


+ Web Services 


l 
l 
l 
l 
l 
l 
Moore Marine i Local 
| Environment 
| 3 
l 
l 
l 
l 
l 


Microbial Project 


WEB PORTAL 


Direct 


Web | 


(other service 


Access 


NASA and NOAA 


Satellite Data Lambda 


| 
l 
| 
| 
| 
Cnxns | 
| 
| 
| 
| 
| 


Community Microbial 
Metagenomics Data 





Local 
Cluster 


TeraGrid: Cyberinfrastructure Backplane | د مس‎ r 





(scheduled activities, e.g. all by all comparison) 
CE 
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Component Sizing 


€ ~ 1000 CPU Cores 
2 512 Cores in now, Another 512 — 1024 in about 1 Year 
€ —225 [B Haw Disk 
> 200 18 now, 400 [ in about year 
€ 10 Gigabit Ethernet as large subsystem interconnect 
2 GigE for all Nodes 
> IB on % of Cluster Nodes 
> Force10 E1200 Switch/Router 


+ Database servers — as needed 
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~ CAMERA Main Page 


WWW.camera.calit2.net 


~~ rm vm. 


Camera: CD 


(Jj http://www.camera.calit2 .net/ vp 


Pm, 


wt camera 


Manne er rer? Ecology 





HOME ABOUT CAMERA METAGENOMICS RESEARCH EVENTS DISCUSSION FORUMS 








WELCOME TO CAMERA! Funded by the Moore Foundation 
Gordon and Betty 
| 
— CAMERA stands for Community Cyberinfrastructure for Advanced Marine =) MO ` RE 
METAGENOMICS 2008 Microbial Ecology Research and Analysis. The aim of this project is to serve = 
November 3 - 7. 2008 the needs of the microbial ecology research community by creating a rich, 
SEES did distinctive data repository and a bioinformatics tools resource that will address What's New 


Metegenomics 2008 [7 many of the unique challenges of metagenomic analysis. 


Have you seen the: 
Version 1.2.6 has been released. 


m— We've added 6 new datasets to our data collection. Some of + New Version 1.2.6 
a rer m /# the key additions are: + Simplified forums 
e a P - Mediterranean Gutless Worm Metagenome « New BLAST user guide 


- Acid Mine Drainage Metagenome 
- Waseca County Farm Soil Metagenome 





Subscribe to the CAMERA mailing list + New Supporting Literature 





to receive notifications of new - and more ... e New information about citing 
software releases and data downloads CAMERA 
as they become available We've also added some improvements: 
9 Upcoming Events 
e Annotation of all metagenomic datasets with new annotation pipeline P 9 
redenen e Improved navigation: more direct access to functions and login no longer + Prochlorococcus 20th 
required to view projects and publications Anniversary Symposium 
e Improved BLAST services: support for large jobs, new export formats, May 30th, 2008 
and rerun of stored job parameters + 108th ASM General 
e Enhanced Fragment Recruitment Viewer: recruitment of metagenomic Meeting June 1st, 2008 
reads to over 1,200 reference genomes with addition of new filters and 
Read this FREE online! H 
Full Book لي‎ | PDF Summary لي‎ Read the version 1.2.6 release notes for more details of the release. Y 


Done 
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CAMERA Portal Login 


web6.camera.calit2.net 


EA £^ FA 


Camera: Login CD 


(UJ https: //web6.camera.calit2.net/gamasso/gamasso?returnURL=http%3A%2P%2Fweb.camera.calit2.net%2Fcameraweb%2Fgwt%2Forg.jcvi.camera.web.g & ¥ Le 


PT, 


et camera — mus 


ÆCONTACTUS -LOGIN 
Manne Mecroteal Ecology 













ABOUT CAMERA METAGENOMICS RESEARCH EVENTS DISCUSSION FORUMS 


Please enter your username and password to login: 


vsemane mj ] 
mn 


Not a registered CAMERA user? Request a CAMERA account 


Forgot your password? Reset it here 


Done web6.camera.calit2.net ف‎ / 
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BLAST Portal 


web.camera.calit2.net 


C OADM CAMERA Research Home 


€ http: / /web.camera.calit2.net/cameraweb /gwt/org.jcvi.camera.web.gwt.home.Home /Home.htm?# 





4 
(8 H 





PR, 


4 


WE camera research 


Manne Mecrobeal Ecology : ‘CONTACT US 











SEARCH 


tooo, A ee 








ABOUT CAMERA METAGENOMICS RESEARCH 


>> Research > Research Hon Research Home Tools # Projects & Data Y 


— Research Home |— ? ^s» 


Fragment 
Recruitment 


Projects Publications 








Recent BLAST Results |— 12) —‏ ل 


Help desk | Disclaimer | Version 1.2.6 
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Projects Include 
postgres SQL database 


000 Browse Projects 
© http: / /web.camera.calit2.net/cameraweb /gwt/org.jcvi.camera.web.gwt.download.BrowseProjectsPage /BrowseProjectsPage.oa 


32 


4 > 


a Camera »»research CONTACTUS -LOGOUT 





للا 


e 











OME ABOUT CAMERA METAGENOMICS RESEARCH NEWS EVENTS DISCUSSION FORUMS 
>> A irch > Projects & Data > Browse Project Research Home Tools Y Projects & Data Y 
el Projects } 
Details 
Global Ocean Sampling Expedition 
m Acid Mine Draina > i 
Metagenome ge Principal Investigator: J. Craig Venter More Information 
Funded By: Moore Foundation, DOE, Venter Science Foundation 
m Alvinella pompejana Epibiont Organization: J. Craig Venter Institute »» Samples 
Metagenome Affiliation: »» Publications & Data 
»» Website 
m Chesapeake Bay The broad objective of the global ocean sampling expedition is to expand our understanding of the >> Contact El 
Virioplankton Metagenome microbial world by studying the gene complement of marine microbial communities. Marine microbes 
(Data Coming Soon) influence the cycling of carbon (and other elements) in the world's oceans, acting as a biological 


conduit that transports carbon dioxide from the surface to the deep oceanic realms. 





m Global Ocean Sampling 
Expedition 





By sequestering carbon from the atmosphere, marine microorganisms (eukaryotes, prokaryotes and viruses) may | 
significantly affect global climate. How they do so, however, is poorly understood, and our attempts to study their 


m Mediterranean Gutless Worm activities are limited by our inability to culture the vast majority of them. 


Metagenome One avenue of exploration is to sequence 


the genomes of marine microbes using a 
metagenomics approach. In Spring of 
2003, the J. Craig Venter Institute 
conducted a whole environment shotgun 
sequencing project to study marine 
microorganisms in the nutrient-poor 


m Mediterranenan Bathypelagic 
Habitat Metagenome 








m Metagenome of Marine 
NaCl-Saturated Brine 


Sargasso Sea near Bermuda. This study M : am gr or D | 


revealed an unforeseen breadth and depth | e 








m Microbial Community 


Genomics at the HOT/ALOHA of microbial diversity - about 1,800 different 


microbial species encoding over 1.2 million 

genes were discovered, nearly doubling 

the number of prokaryotic genes available 

in public databases. Notably, this study expanded our knowledge of 
ocean photobiology and nutrient pools. Results from the pilot study 
were reported in Science in 2004. 


m Moore Marine Microbial 
Sequencing 





m Ocean Viruses 





This pilot study served as the springboard for launching a more 
z pe mmm | — ES PO iP / comprehensive survey of the bacterial, archaeal and viral diversity of 
np the world's oceans. À alobal circumnaviaation aboard the Sorcerer Il 














Dananrah Maren 


Done 
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Download Data 
ftp.camera.calit2.net 


Browse Publications and Data € 









(Jj http://web.camera.calit2 .net/cameraweb /gwt/org.jcvi.camera.web.gwt.download.DownloadByPubPage /DownloadByPubPage.oa# ve 
M 
Pm, 
P -— 
“Camera m 
>> research Re CONTACTUS Loco M 


Manne Mwcrobal Ecology 








HOME ABOUT CAMERA METAGENOMICS RESEARCH NEWS EVENTS DISCUSSION FORUMS 
>> | irch > F &I »P ition id I Research Home Tools Y Projects & Data Y 








— Publications and Data | 


Project: | Global Ocean Sampling Expedition = 


Details Downloads 








[34 width] 
m The Sorcerer Il Global Ocean Sampling Expedition: The Sorcerer Il Global Ocean Sampling Expedition: Northwest Atlantic through Eastern Tropical Pa 
Northwest Atlantic through Eastern Tropical Pacific 
Douglas B. Rusch, Aaron L. Hal... Publication Downloads 
= Structural and functional diversity of the microbial Publication 
kinome ss 


Natarajan Kannan, Susan S. Tay... 
Publication Data Downloads 

m The Sorcerer Il Global Ocean Sampling Expedition: 

Expanding the Universe of Protein Families Filters [show] 

Shibu Yooseph, Granger Sutton,... 

[expand all] [collapse all] 

m Environmental Genome Shotgun Sequencing of the - Site Metadata 

Sargasso Sea ; 11 KB) 

J. Craig Venter, Karin Remingt... Ej GOS Sites (11 KE 


[3 Chesapeake Bay NSF MOVE (4 
m Assessing diversity and biogeography of aerobic 








anoxygenic phototrophic bacteria in surface waters - Reads 

of the Atlantic and Pacific Oceans using the Global D All GOS Reads (13 GB) 

Ocean Sampling expedition metagenomes 

Natalya Yutin, Marcelino T. Su... [3 GS000a Shotgun, Sargasso Sea, Station 13 and 11 (1.1 GB 

n Gene Identification and Protein Classification in [5 35000 Shotgun, Sargasso Sea, Station 13 and 11 

Microbial Metagenomic Sequence Data via [3 GS000c Shotgun, Sargasso Sea, Station 3 (650 MB) 
Incremental Clustering eR 
Shibu Yooseph, Weizhong Li, Gr... [3 GS000d Shotgun, Sargasso Sea, Station 13 (599 MB) 

[3 GS001a Shotgun, Sargasso Sea, Hydrostation S (250 MB) 

m Viral Photosynthetic Reaction Centre Genes and 
Transcripts in the Marine Environment B 35001 Shotgun, Sargasso Sea, Hydrostation S (160 MB) 

———— [3 GS001c Shotgun, Sargasso Sea, Hydrostation S (160 MB) 

m The Sorcerer Il Global Ocean Sampling Expedition: [3 GS002 Shotgun, Gulf of Maine (150 ME) - 
Metagenomic Characterization of Viruses within Ra EVE £ 
Aquatic Microbial Samples [3 GS003 Shotgun, Browns Bank, Gulf of Maine MB) i 

C 7 "a k 


Done 
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User Forums 


web7.camera.calit2.net 








5/15/08 
















000 CAMERA Forum 6 
Ki http://web7.camera.calit2.net/jforum/forums/list.page NAAN 
at! 
Le Wi 
Camera 
Pig RTE سه‎ Search  —— | go | 
ABOUT CAMERA METAGENOMICS RESEARCH EVENTS DISCUSSION FORUMS 
AQ search O Recent Topics (E Member Listing [& My Profile D My Bookmarks |2! Private Messages 
You last visited on: 03/25/2008 16:23:11 
The time now is: 04/23/2008 14:25:44 
Forum Index Read new messages since my last visit 
Forums 
Known Bugs 04/18/2008 18:00:03 
Known bugs, problems, and service downtime notices. abrust +D 
Known Data Issues 03/20/2008 10:31:19 
Known and reported issues concerning consistency and quality of CAMERA datasets. pgilna *D 
Report A Problem 04/07/2008 11:17:58 
Report problem with CAMERA system or applications. yoyoman +D 
Make A Suggestion 05/31/2007 05:53:20 
Provide feedback on new features or suggestions on improving the CAMERA services. CNRS +D 
Let The Community Know 01/24/2008 15:12:30 
Provide information to the Marine Metagenomics community. genbio *D 
Who is online 
Our users have posted a total of 136 messages 
We have 374 registered users 
(26) The newest registered user is crabtree 
There are 1 online users: 1 registered, 0 guest(s) [ Administrator ] [ Moderator ] L 
Most users ever online was 254 on 11/13/2007 03:12:18 A 
Connected users: mjk C 
Y 
4 


Done 
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-= BLAST Query 


searching for ecoll 




















EEE B 
66 BLAST Wizard C 
(Jj http://web.camera.calit2 .net/cameraweb /gwt/org.jcvi.camera.web.gwt.blast.Blast/Blast.htm&BlastWizardUserSequencePage vii 
M 
Pm, 
P P. sd 
` Camera e — 
| >> research ‘conracrus oc ou" SE 
Manne Mucroteal Ecology 
HOME ABOUT CAMERA METAGENOMICS RESEARCH NEWS EVENTS DISCUSSION FORUMS 
>> | irch > Tools > BLAST 5 > Pa Research Home Tools Y Projects & Data Y 
el Specify Query Sequences ل‎ ? help 
Specify a nucleotide or peptide query sequences (in multi-FASTA format El): 1 Next i 





New Sequences Previous Sequences 


Sequences Name: 
User sequence 04/23/08 02:15 PM 


© Enter sequences in multi-fasta format: 


agcttttcattctgactgcaacgggcaatatgtctctgtgtggattaaaaaaagagtctctgacagcagcttctgaactgqt 





^ Upload multi-fasta sequences file 


| Browse... | ( Upload ) 





Done 
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Choose Dataset 
~ BTB FASTA files & SQL DB 





aaa BLAST Wizard 


© http://web.camera.calit2.net/cameraweb/gwt/org.jcvi.camera.web.gwt.blast.Blast/Blast.htm#BlastWizardSubjectSequencePage v 
















Read Datasets Í Assembled Datasets [ External Datasets | All Datasets 


Description 

325,778,024 | nucleotide 
179,275,878 | nucleotide 
33,370,426 | nucleotide 





4,439,529,911 


peptide 





13,318,589,733 


nucleotide 





| 12,132,838,728 | 


nucleotide 





290,371,756 | nucleotide 

7,203,198 | nucleotide 
221,211,059 | nucleotide 
154,475,569 | nucleotide 

2,380,900 | nucleotide 
GOS: All Metagenomic Sequence Reads (N) 10,634,523,014 | nucleotide 
GOS: All ncRNAs (N) 33,226,207 | nucleotide 























4,395,420,221 | peptide 

13,186,260,663 | nucleotide 
314,746,819 | nucleotide 
63,837,557 | nucleotide 
144218 | 





























Cale? 


nucleotide 











( Back | ( Next | 





حم 
Le‏ 
a‏ 
pis‏ 
camera >> research i "CONTACT US‏ 
Morine Microbial Ecology‏ 
HOME ABOUT CAMERA METAGENOMICS RESEARCH NEWS EVENTS DISCUSSION FORUMS‏ 
R ich > Tools > BLAST Sequences > Page 2 of Research Home Tools ” Projects & Data Y‏ >> 
el Select Reference Datasets to Search Against + 7) help‏ 
4 
y‏ 


Help desk | Disclaimer | Version 1.2.6 





Done 


5/15/08 © 2008 UC Regents 143 





Name Your Job 















BLAST Wizard C 

(Jj http://web.camera.calit2 .net/cameraweb /gwt/org.jcvi.camera.web.gwt.blast.Blast/Blast.htm#BlastWizardSubmitjobPage v è 

al 
Pm, : 

; camera wt 
V »»research 'contaerus Locout DE 
Morine Microbial Ecology 
HOME ABOUT CAMERA METAGENOMICS RESEARCH NEWS EVENTS DISCUSSION FORUMS 
>>R irch > Tools > BLAST Seguen > Pag f Research Home Tools Y Projects & Data Y 


cl Job Criteria } 


Job Name:" 
Query Sequence: User sequence 04/23/08 02:15 PM [change] 
Subject Datasets: GOS: All Metagenomic Sequence Reads (N) [change] 








el Job Options - ? help 


Program: | BLASTN (nuc/nuc) x 





Basic Options 
db alignments per query: [25 evalue exponent (1Ex): Z 
filter low-complexity seq: T | lower case filtering: € True False 





Advanced Options [show] 











| Back | | SubmitJob | 
Help desk | Disclaimer | Version 1.2.6 


5/15/08 © 2008 UC Regents 144 





Hunning 
oun Grid Engine Job 


(Jj http://web.camera.calit2 .net/cameraweb /gwt/org.jcvi.camera.web.gwt.blast.Blast/Blast.htm#SubmitjobWaitPage 








pm, 


^ Camera »»research 1 "CONTACT US 


Morine Microbial Ecology 





EVENTS 






p: 
9 SEARCH 


LOGOUT EH 


DISCUSSION FORUMS 





METAGENOMICS RESEARCH NEWS 


HOME ABOUT CAMERA 
3 F Research Home Tools # Projects & Data Y 


>> | | > To > BLAST | > Run 


Your reference sequence is being aligned to your subject sequences. 


Your job number is 1204962519605577343. 


You can either wait here for the job to complete, or use the menus above. 
Your job results will be available on the Job Results page 


Help desk | Disclaimer | Version 1.2.6 


EE Jar 
E 
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. Waiting... 


(Job running on cluster) 


Job Results 








ANDO 
| (Jj http://web.camera.calit2 .net/cameraweb /gwt/org.jcvi.camera.web.gwt.status.Status /Status.htm&JobResultsPage 








"CONTACT US 





SEARCH 
oso, rn 









oi lp 
^m 
© Camera 
>> research 
Marne Mecrobsal Ecology 
HOME ABOUT CAMERA METAGENOMICS RESEARCH NEWS EVENTS DISCUSSION FORUMS 
Tools > BLAST Results Research Home Tools Y Projects & Data Y 
prev 2J' | goto » | next zu 


>> Research > 


— BLAST Results |— [hide] 7 helo 
Advanced Sort 


1-5 5 


bixon isan [0921070225 AM completed B 


Show: 10 20 50 


ecoli-reads [edit] | 04/23/08 02:17 PM | running né blastn | 


of 5 


un 


1- 


Job Name Submit Date ¥ Status Program # Hits Subject Sequences 
GOS: All Metagenomic Sequence Reads (N) 
GOS: Assembled Sequences (N) 


GOS: All Site-specific Assembled Sequences (N) 
All Metagenomic Sequence Reads (N) 


GOS: All Metagenomic Sequence Reads (N) 


Help desk | Disclaimer | Version 1.2.6 
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Done 
































aag Job Results C 
g http: / /web.camera.calit2.net/cameraweb /gwt/org.jcvi.camera.web.gwt.status.Status /Status.htm&JobResultsPage vi 
pi” 
€ camera 
>> research 
HOME ABOUT CAMERA METAGENOMICS RESEARCH NEWS EVENTS DISCUSSION FORUMS 
>> Research > Tools > BLAST Results Research Home Tools Y Projects & Data Y 
BLAST Results [hide] ? help 
1-5of5 Advanced Sort e oto Y ext 2 
Job Name Submit Date 4 Status Program # Hits Subject Sequences Actions 
ecoli - reads {edit} | 04/23/08 02:17 PM | completed blasn | 25 GOS: All Metagenomic Sequence Reads (N) | Results / | Job * | / | Expon + | 
04/23/08 02:02 PM completed | blastn | 25 | GOS: Assembled Sequences (N) Results / | Job * | / | Export + | 
| X | bkk 01 [edit 03/21/07 02:25 AM completed | blastn | 0 | GOS: All Site-specific Assembled Sequences (N) ( [sob ] / [Expo 
X | from bangkok [edit] | 03/21/07 01:56 AM | completed blastn | 25 | All Metagenomic Sequence Reads (N) Results / | Job | / | Export + | 
X | from sofitel fedt] — 03/12/07 05:42 PM | completed | blastn | — $ GOS: All Metagenomic Sequence Reads (N) | Results / | Job +] / | Export + | 
1-5of5 Show: 10 20 50 « previ oto ¥ ext لاك‎ P> 
c 
à 
Help desk | Disclaimer | Version 1.2.6 e 
Jar 
Done L 
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Results 


ex m" rm 


Job Results 





© http://web.camera.calit2.net/cameraweb/gwt/org.jcvi.camera.web.gwt.status.Status/Status.htm# 


zw, 





^ 


6 


4 


Manne Maroteal Ecology “CONTACT US ‘LOG OUT 











ABOUT CAMERA METAGENOMICS RESEARCH NEWS EVENTS DISCUSSION FORUMS 


>> | >T > EL/ f > BL Deta Research Home Tools Y Projects & Data Y 


m Job Summary ا‎ [hide] a back to job results > help 


E Camera »»research "— SEARCH 









































Job ID: 1204962519605577343 Submitted: 04/23/08 02:17 PM Query Sequence: User sequence 04/23/08 02:15 PM Show parameters: [view parameters] 
Job Name: ecoli -reads Program: blastn Subject Sequence: GOS: All Metagenomic Sequence Reads (N) 
d Matching Sequences ل‎ [show] , p d Sequence Geography = [hide] 
+ Sequence Alignment } [hide] ad 
PA FA ES ^ : TRES 
Sequence: JCV| READ 396311 Score: 121.416 identities: 70/73 (95%) EE KER Galal SEE 
aa Mési 
Sequence Length: 950 Expect: 8.36055e-25 Positives: 0/73 (0%) VE 
Alignment Length: T3 Query Begin/End: — 128-200 (Minus) Query Gaps: 0 3 
Clear Range: 69-778 Subject Begin/End: 375-447 (Plus) Subject Gaps: 0 te Guatemala 


T TÉ 5 
ndi Pi 
ATGGATGTTGTGTACTCTGTAATTTTTATCTGTCTGTGCGCTATGCCTATATTGGTTAAAGTAT 128 
ODE OE EDEN 

ATGGATGTTGTGTACTCTGTAATTTTTATCTGTCTGTGTCCTGTGCCTATATTGGTTAAAGTAT 447 


Query: 200 WHT 
| | | 


Roe yg, Venezuela". 
Sbjct: 375 ATGCGTTTC | 


kent ‚Guyana 
8 58 : "| Pre 4 
Colombia. me 


Key: match similar 1 stop codon 














>> 14 samnile sites ara ranracantan in thic data cat 





Done 
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Download Datasets 


simple http server 





5/15/08 





CAMERA 





© http://web4.camera.calit2 .net/files/ 


pm, 





Marine Mecrobeal Ecology 


ABOUT CAMERA METAGENOMICS 





ÆCONTACTUS -LOG OUT 








RESEARCH EVENTS DISCUSSION FORUMS 


SEQUENCE DOWNLOADS 


CAMERA genomic data sets are listed below. All data is compressed using gzip. 


node1015438648499241313.fasta 


Name: GOS: Combined Assembly Proteins (P) 


Description: this file contains proteins predicted via the clustering 
approach as well as proteins with HMM matches 


Sequence Count: 6109770 

Sequence Type: peptide 

Length: 1,225.15 MB 

Tags: combined_gos,predicted_protein 
Dataset Node ld: 1015438648499241313 


node1015438648499241314.fasta 


Name: GOS: Combined Assembly Coding Sequences (N) 


Description: this file contains nucleotide coding sequences 
corresponding to the proteins predicted via the 
clustering approach as well as proteins with HMM 
matches 


Sequence Count: 6109770 

Sequence Type: nucleotide 

Length: 3,666.29 MB 

Tags: combined_gos,predicted_gene 
Dataset Node Id: 1015438648499241314 


Available Files 


node101543864849924 1313 fasta.gz 783.8 MB 2008-03-12 12:58:26 
node1015438648499241313 fasta.gz.md5 67 bytes 2008-03-12 12:58:26 
node1015438648499241313.info 337 bytes 2008-03-12 12:58:26 


Available Files 


node101543864849924 1314 fasta.qz 13GB 2008-03-12 12:58:44 
node1015438648499241314 fasta.gzmd5 67 bytes 2008-03-12 12:58:44 
node1015438648499241314 info 394 bytes 2008-03-12 12:58:44 








Done 
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Logical Layout of Servers o> 


Pe = 
‘Single Bin On Laser | 0 M | 


Web 
Server 


Cluster 
Frontend 


ClusterNôdées and File Servers 





2 (cH 





Portal 
Server 






(Tomcat) 


Public Net 
Private Net 


Postgres 


Tä Database 


(Jboss) 





© 2008 UC Regents 


Ki Portal/Database/Blast Server 
Interactions 





+ Portal is a Tomcat Java 
application. (camera-app 
appliance) 

2 All data presented in the portal is 
kept in a Postgres Database 

€ When a BLAST job is submitted, 
control of the job is handed to a 
JBOSS application (Camera- 
Compute) 

+ The Camera-Compute formats a 
batch job and submits to the 
cluster frontend via DRMAA 
interface 


Web Portal 
Tomcat / Java 
camera-app 
Database 
Postgres 
postgres8db 
BLAST Master 
JBOSS / Java 
camera-compute 
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Compute Server Segmenting 


Job Scheduler 
Sun Grid Engine 
frontend 
Database 
Postgres 
postgres8db 
BLAST Master 
JBOSS / Java 
camera-compute 


5/15/08 © 2008 UC Regents 152 





+ Blast Master receives job 
request from Portal Server 
Reads from Postres Database 

how Blastable files have been 
segmented 
> All ORFs — 134 Segments 
> All Reads — 84 Segments 
€ An SGE Array Job is send to 
the queue master for 
submission to compute cluster 
+ Raw Results are stored in the 
file system 



















CAMERA System Architecture. A Modestly 
Complex Grid Endpoint 















NAS 
Multi-TB Storage (NSF) 
GOS Data Master Copy 





Logical Server Group 
Interdependent 


5/15/08 © 2008 UC Regents 153 





e Using Hocks to Manage 
Complete Deployment 


€ Production CAMERA is deployable by one system 
engineer 
€ We have to support three phases of software 
development 
2 Development 
2 Staging (Testing) 
2 Production 
+ Development is a done on a completely separate 
cluster 
2 Development in rockscluster.org domain 
2 Production/Staging in camera.calit2.net domain 


2 All share the same GAMA (Certificate Authority) server 
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Is Break Apart Software Stacks into Logical 






Hol 





L3 A 









Applications 
HEHEH]! 
Message Passing / Communication Layer 
Linux Environment 
Kernel 
Linux Kernel | 





Rocks 
Base 





T " 
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A 


CO COCO OUT OO CHE IEEE‏ لال 


(LIU CLLEL LENS U 


i e Í 
LOU EU Q LOU LUUR 





"TTTTTT] TO TEEN DENNE 


Standard 


Compute 





Configuration 
Graphs 


Describe how to 
puild a complex 
of clusters 
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e Why CAMERA is not too 


AMERA 7 


C 
Complex 


| 








Rolls 








22000 

















cameras D 
/ i 
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5 im 


die grep 


| 
i 
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CAMERA Graph In Detail 








How the Different Appliances 
Are Defined 


CD ` CMD + Appserver 

CD Gd 5 FTP server 
Np App server Firewall 

Tomcat Server 

Bio Roll Base 

Client 


+ Camera Compute Server 
> SGE Client 
> Jboss Server 
> Bio Roll Base 
> Client 


Cluster Compute Node 

> Bio Roll Base 

> Camera Compute addons 

> SGE Client 

> Compute-Appliance > Client 
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OO O 0 





e$ Inside the Camera-app-server - 


Packages 


<kickstart> 
<package>cameraweb</package> 
<package>cameraweb-jaas</package> 
<package>gama-sso-client</package> 
<package>postgres8</package> 
<package>libungif</package> 
<package>fonts-xorg-75dpi</package> 
<package>Xaw3d</package> 
<package>emacs-common</package> 
<package>emacs</package> 
<package>emacs-el</package> 
<package>emacs-nox</package> 


</kickstart> 


3 Camera Specific 


+ One generic, but 
different version 


+ Remainder are 
Standard RedHat 
packages 
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e$ Inside the Camera-app-server - 
Config 


cht eh EK 
«file name="/etc/fstab" mode-"append"» 


<var name-"GAMASSO filestore-server"/>:/var/tmp/gama /var/tmp/gama nfs 
ro, soft, actimeo=3 i Bl 


Feb Ex 


üomN/chkconfig --add gama-sso-client 


«file name="/opt/tomcat/webapps/cameraweb/WEB-INF/classes/camera.properties"> 
<eval> 

file=/opt/tomcat/webapps/cameraweb/WEB-INF/classes/camera.properties 
/opt/rocks/bin/rocks list camera javaprops $file 

«/eval» 

«/file» 


chown 412.412 /opt/tomcat/webapps/cameraweb/WEB-INF/classes/camera.properties 


* Mount the single-sign-on file system from the SSO server 
٠ Localize the Cameraweb java application 
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Ged How to Handle Java Application 
Location 


+ Many Java applications have properties files that are 
read to define localization 
> Where to find files 
2 Location of support servers 
2 Debug level 
€ The CAMERA Blast Portal has over 50 localized 
variables 
2 Editing manually can lead to higher error rates 
+ Variables are different for Production, Development, and 
otaging 
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e Camera.properties File 
Rewriting 


Original file with paths defined in developers environment 





# DRMAA Blast Merge/Sort settings 

BlastServer.GridJavaPath-/usr/local/bin/java 

BlastServer.GridMergeSortClassPath-/project/camera/runtime-shared/1.0.x/jars/ 
camera-blast-grid-merge.jar 

BlastServer.GridMergeSortProcessor-org.jcvi.camera.shared.blastxmlparser.Blast 
GridMergeSort 

BlastServer.GridMergeSortMinimumMemoryMB-384 


Deployed file with paths defined for production environment 


# DRMAA Blast Merge/Sort settings 

BlastServer.GridJavaPath-/usr/java/jdk1.5/bin/java 

BlastServer.GridMergeSortClassPath-/opt/camera/camera-blast-grid-merge.jar 

BlastServer.GridMergeSortProcessor-org.jcvi.camera.shared.blastxmlparser.Blas 
tGridMergeSort 


BlastServer.GridMergeSortMinimumMemoryMB-384 
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e Generic Properties (Not Just 
Java) 


+ The Rocks Database is the Master Record for variables 
2 Any Variables set in a configuration file are defined here 

+ Key, Value table (app globals) has four critical fields 
2 Appliance, Service, Key, Value 

€ For example 


20 (0 = valid for any appliance, <0 for a particular appliance 
only) 


> lopt/tomcat/webapps/cameraweb/WEB-INF/classes/ 
camera.properties 


> BlastServer.GridJavaPath 
> «indirect service="CAMERAETC" component="JavaPath"/> 
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<indirect> Construction 


+ In Building CAMERA we found that 
2 Many different (non-Rocks) components needed the identical value 
> EJ. 


Cameraweb application needed name of SSO server 
{etc/fstab also needed name of SSO server 
Cameraweb needed name of Database server and connection info 


- So does the Jboss Compute server 
- So does an Administrative (read-only) console 
2 Production/Staging environments required small changes 


Staging Server needed it a Staging (test) database 
Most other variables were identical with production version 


€ <indirect> 
> Inthe "value" part of the database. 
2 Acts a data pointer to another [Service, Key, Value] triplet 


2 When a non-zero appliance is supplied, can overwrite for the generic 


version 
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Using <indirects> 


+ Rocks de-references all indirects at kickstart generation — application/roll developer need not know 
about indirects. 
+ Available as «var name-"Service Component”> 
>  Appliance-specific overrides are handled automatically 
€ From appserver definition 


<var name="GAMASSO filestore-server"/>:/var/tmp/gama /var/tmp/gama nfs ro,soft,actimeo-3 
TT 
«/file» 
= EE Iii NS eee eeen + 
| service | Component | value | 
TETEE Im —————— | ————A—R————A—————AA——————————— + 
| GAMASSO | filestore-server | «indirect service="GAMASSO" component="PrivateHostname"/> | 
| GAMASSO | Debug | false | 
| GAMASSO | FileDirectory | /var/tmp/gama | 
| GAMASSO | CookieName | gamasso | 
| GAMASSO | PrivateHostname | «indirect service-"CAMERAHOSTS" component="SSOServerPrivate"/> | 
| GAMASSO | SiteDomainname | camera.calit2.net | 
ES rae ee HR E (We = eee eeen + 


+ Filestore server is double indirect as 
2 CAMERAHOSTS, SSOServerPrivate 
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Production and Staging Hosts 


+------------ سس سس سس سس سي‎ +----------------------- +------------------------------- + 
| Membership | service | Component | value | 
+------------ +------------- +----------------------- +------------------------------- + 
| O | CAMERAHOSTS | AppServer | web.camera.calit2.net | 
| 13 | CAMERAHOSTS | AppServer | web3.camera.calit2.net | 
| 14 | CAMERAHOSTS | AppServer | web3.camera.calit2.net | 
| O | CAMERAHOSTS | ComputeServerPrivate | computeserver-0-0 | 
| 13 | CAMERAHOSTS | ComputeServerPrivate | computeserver-test-0-0 | 
| 14 | CAMERAHOSTS | ComputeServerPrivate | computeserver-test-0-0 | 
| O | CAMERAHOSTS | ForumsServer | web7.camera.calit2.net | 
| O | CAMERAHOSTS | GamaServer | gama-camera.rocksclusters.org | 
| O | CAMERAHOSTS | GridSphereServer | web/.camera.calit2.net | 
| ON E GAMER ARGS TS || ١ 8 sit Gres Sem ve sP mi uwascH M Post ere se db s0 ss | 
| 13 | CAMERAHOSTS | PostgresServerPrivate | postgres8db-0-0 | 
| 14 | CAMERAHOSTS | PostgresServerPrivate | postgres8db-0-0 | 
| QO | CAMERAHOSTS | PRWebServer | camera.calit2.net | 
| O | CAMERAHOSTS | SSOServer | web6.camera.calit2.net | 
| 0 | CAMERAHOSTS | SSOServerPrivate IB ott Hp 2 ce al. | 
+------------ +------------- ست سس ست سس سس سس سس سيل‎ +------------------------------- + 
13 == Appserver Testing Appliance 

14 == Computeserver Testing Appliance 
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Summary of Layout 


+ Different servers must be applied as a group 
> [application, camera compute, database, SSO] 
2 We have production and staging (test) versions 


+ Configuration variables are held in a database 
> Data “pointers” are supported using «indirect» 


2 Hocks Kickstart framework dereferences all «indirect» 
statements 


€ loday, there are 142 direct and indirect references in 
our Configuration Database 


> 72 of these are «indirects» 
5/15/08 © 2008 UC Regents 167 





e 


Viz & CAMERA Taught Us 


e Viz and CAMERA helped Rocks 
2 Better abstractions in XML 
2 Learned new ways to use the existing 
framework 
+ We were surprised at 


2 How small the differences are between Viz, 
CAMERA, and HPC Clusters 


> How well our framework supported such 
functionally different systems 
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